Add adaptive circuit breaker by AbdulRahmanAlHamali · Pull Request #760 · Shopify/semian

AbdulRahmanAlHamali · 2025-09-23T15:16:41Z

Add a new adaptive circuit breaker that:

Works without need for configuration
Has the ability to open partially/fully depending on the severity of the incident

The circuit breaker has two main components:

Ideal Error Rate Estimator

This estimator tries to find out the expected error rate from a healthy dependency. It does that through a method of simple exponential smoothing, which basically relies on calculating a weighted average, with a few extra domain-specific hints:

Ignore any observations that are obviously too high
Converge slowly towards higher values, and quickly towards lower values
Be more receptive to signals for the first 30 minutes after boot (because we start with a random guess for the value), and less receptive afterwards

PID Controller (Process, Integral, Derivative) Controller

See wikipedia link

This controller increases/decreases the the rejection rate of the circuit breaker based on the value of:

kI * Integral + kP * P + kD * derivative

Where P is:

ErrorDelta - (1 - ErrorDelta) * RejectionRate

The intuition of the value of P:

Increases when error rate increases
Decreases when rejection rate increases
The rejection rate has a (1 - ErrorDelta) multiplier, that we call the "defensiveness" multiplier. This allows rejection to increase more aggressively if the error rate is high

Note: The derivate component is currently set to 0. The Integral component mostly contributes as a history to prevent the circuit breaker from fluctuating too much.

kris-gaudel · 2025-09-23T18:08:00Z

lib/semian/adaptive_circuit_breaker.rb

+    attr_reader :name, :pid_controller, :ping_thread
+
+    def initialize(name:, kp: 1.0, ki: 0.1, kd: 0.0,
+      window_size: 10, history_duration: 3600,


Regarding the calculation of online mean, Welford's algorithm was discussed as a solution to use constant space

Also, do we want to allow developers to configure the interval of integration? (history_duration)?

Update variable names Fill sliding window with 1 hr worth data Add comment Update experiemnt resource to be deterministic Change deterministic default value to false Cleanup Remove unused variable Make initial seed error rate more customizable Add seed_error_rate as a property

* Prefilling added * Change initial duration to 900 s

* testing different circuit breaking scenarios * adding concurrency * adds more puts to get further information during phases * Fixing concurrency, unprotected ping, extras * update classic sustained test * cleaning up outdated tests, and testing without ping rate * modify ki instead of dividing by window size --------- Co-authored-by: Abdulrahman Alhamali <abdulrahman.alhamali@shopify.com>

* allow using the fiber scheduler if present * run experiments

…oved calculations

…rror-rate-calculation Fixed redundant ideal error rate calculation. Calculated once and re-used appropriately.

* Added param to only use 'current_window_requests' when needed. Updated 'start_pid_controller_update_thread' to only pass in needed values to 'notify_metrics_update'. * Removed notifying logic for state transitions. * Refactor adaptive circuit breaker tests by removing the state transition notification test. Updated metrics expectations to include previous p_value and adjusted test logic for clarity. * Added updated experiments

kris-gaudel · 2026-01-20T03:21:26Z

Hey guys, hope you had a good start to the new year! Are there any issue I could work on in an open-source capacity on the ACB?

samuel-williams-shopify · 2026-01-20T21:45:47Z

lib/semian/adaptive_circuit_breaker.rb

+    end
+
+    def wait_for_window
+      Kernel.sleep(@sliding_interval)


Suggested change

Kernel.sleep(@sliding_interval)

Kernel.sleep(@sliding_interval * rand(0.9...1.1)

samuel-williams-shopify · 2026-01-20T21:49:40Z

Here are the notes from our tuple:

(A) 1 PIDController
(B) 1 Faraday request

(A) -> sleep(sliding_window_timeout)
(B) -> timeout(t, IO#write -> IO#gets)
(A) -> wakes up, update
(B) -> timeout(t, IO#write -> IO#gets -> response line) -> good

(An) -> sleep(sliding_window_timeout + random_jitter)
(A0) -> wakes up, update -> some CPU time, -> A(N+1) -> time has elapsed
(B) -> timeout(t, IO#write -> IO#gets)
(AN+1) -> wakes up, update # process all of these O(An)
(B) -> timeout(t, IO#write -> IO#gets -> response line) -> times out


Assume sliding window of 1 second
(A0) t+0.1
(A1) t+0.2
(A2) t+0.3
(A0) -> wakes up, update -> some CPU time, -> A(N+1) -> time has elapsed
(A0) t+0.1 -> ready at t+1.0 -> sleep(sliding_window_timeout)
(A1) t+0.2 -> ready at t+1.0 -> sleep(sliding_window_timeout)
(A2) t+0.3 -> ready at t+1.0 -> sleep(sliding_window_timeout)

(1) +random jitter -> always late
(2) +/- random jitter -> gaussian distribution of lateness
(3) + robust loop which computes offset, but you may need to drop sample windows if running late
	(real time stuff)

duration = 1.0
def now
	Process.clock_gettime(Process::CLOCK_MONOTONIC)
end


while true
	current = now
	yield

	next_time = current + duration
	duration = next_time - now
	if duration > 0
		sleep(duration)
	else
		# droppped window
	end
end

while true
	# uniform distribution:
	duration = duration * rand(0.9..1.1)
	sleep(duration * rand(0.9..1.1))

	yield(duration) -> pid_controller.update(duration)
end

* add jitter to sleep * fix tests and re-run experiments --------- Co-authored-by: Adrian Gudas <adrian.gudas@shopify.com>

…reated

stop thread if the last remaining circuit breaker is destroyed use a sliding interval of 1 (run through the loop every second)

…hread using a single thread for all PID controller statuses

* Add dead zone ratio to PID controller for noise suppression - Introduced `dead_zone_ratio` parameter in the PID controller to suppress noise from small deviations in error rates. - Updated the `calculate_p_value` method to implement dead zone logic, allowing for more stable control responses. - Enhanced tests to validate the behavior of the dead zone in various scenarios, ensuring it does not impede recovery while effectively filtering noise. * Added experiment tests * Refine dead zone logic in PID controller - Updated the handling of the dead zone in the PID controller to allow full signal response above the dead zone, improving control accuracy. - Adjusted comments for clarity regarding the purpose of the dead zone in noise suppression. * Added experiment results

kris-gaudel reviewed Sep 23, 2025

View reviewed changes

AbdulRahmanAlHamali force-pushed the pid-take-2 branch from 4c9eb68 to 550a46f Compare October 12, 2025 22:34

Aguasvivas22 force-pushed the pid-take-2 branch from 69ab5b7 to 550a46f Compare October 13, 2025 13:33

AbdulRahmanAlHamali and others added 24 commits October 22, 2025 17:08

add adaptive circuit breaker

4ef2649

add ability to enable adaptive circuit breaker

b19e198

fix constructor

824f229

address deadlock issues and use standard clamp function

d14c397

add time freeze to tests

fec549a

remove unnecessary stubbing

23f6ec8

cap ideal error rate, and fix tests

ef7ec52

more test fixing

54e036e

use a discrete time window in pid controller

e037ead

add unprotected ping

3f59687

fix bugs and improve example

ef08850

add ability to enable adaptive circuit breaker

4af0551

use a discrete time window in pid controller

9ce68c6

Add P2 estimator

9f4357e

Use P2 estimatori in PID

6795d2e

Add results

e1e055c

Update variable names

2c64138

Interpolate instead of round

6bf05dd

Add missing case

56c1db0

Remove platform

2e43221

Fix cold start issue (#809)

4a6f482

* Prefilling added * Change initial duration to 900 s

Remove outputs

86a733e

AbdulRahmanAlHamali force-pushed the pid-take-2 branch from d52fa4d to 56ed006 Compare October 22, 2025 17:08

AbdulRahmanAlHamali added 2 commits October 22, 2025 17:09

remove unintentional change

90cefeb

remove AI fluff

f96df80

github-actions bot added the cla-needed label Jan 5, 2026

AbdulRahmanAlHamali and others added 5 commits January 12, 2026 19:24

Allow using the fiber scheduler if present (#972)

77cb2b0

* allow using the fiber scheduler if present * run experiments

Update PID controller to store and use last ideal error rate for impr…

7a4c941

…oved calculations

Added CSV and PNG files for all experiments ran.

89d8752

Merge pull request #983 from Shopify/abdullah/fixed-redundant-ideal-e…

d873937

…rror-rate-calculation Fixed redundant ideal error rate calculation. Calculated once and re-used appropriately.

samuel-williams-shopify reviewed Jan 20, 2026

View reviewed changes

AbdulRahmanAlHamali and others added 21 commits January 21, 2026 19:29

Add jitter to sleep (#990)

ebdd1f8

* add jitter to sleep * fix tests and re-run experiments --------- Co-authored-by: Adrian Gudas <adrian.gudas@shopify.com>

initial commit

e39603f

don't use a separate WorkerState object

a753114

use a class variable so that it exists before the class instance is c…

537204f

…reated

re-run experiments

a33e259

clean up terminology

a2e865f

stop thread if the last remaining circuit breaker is destroyed use a sliding interval of 1 (run through the loop every second)

reduce code duplication in dual circuit breaker

97973a9

add experiments

cef49bb

remove the ability to stop the update thread (for now)

d8738ed

attempt to fix tests

53f889d

stop the thread if all circuit breakers are destroyed (fixes tests)

e36461f

make sure initialize doesn't run twice

0a495da

add experiment results

af67ec2

working single thread implementation

43c2b0b

merge in latest metrics optimizations, and fix tests

8837cad

revert changes to pid_controller_test

440f4d0

re-run experiments

2be4606

use fibers if SEMIAN_PID_CONTROLLER_USE_FIBERS is true

8ff192a

add experiments

02d22cb

Merge pull request #973 from Shopify/adriangudas/adaptive-cb-single-t…

6f4df13

…hread using a single thread for all PID controller statuses

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add adaptive circuit breaker#760

Add adaptive circuit breaker#760
AbdulRahmanAlHamali wants to merge 164 commits intomainfrom
pid-take-2

AbdulRahmanAlHamali commented Sep 23, 2025 •

edited

Loading

Uh oh!

kris-gaudel Sep 23, 2025

Uh oh!

kris-gaudel Sep 24, 2025

Uh oh!

kris-gaudel commented Jan 20, 2026

Uh oh!

samuel-williams-shopify Jan 20, 2026

Uh oh!

samuel-williams-shopify commented Jan 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

	Kernel.sleep(@sliding_interval)
	Kernel.sleep(@sliding_interval * rand(0.9...1.1)

Conversation

AbdulRahmanAlHamali commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ideal Error Rate Estimator

PID Controller (Process, Integral, Derivative) Controller

Uh oh!

kris-gaudel Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

kris-gaudel Sep 24, 2025

Choose a reason for hiding this comment

Uh oh!

kris-gaudel commented Jan 20, 2026

Uh oh!

samuel-williams-shopify Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

samuel-williams-shopify commented Jan 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

AbdulRahmanAlHamali commented Sep 23, 2025 •

edited

Loading